Evaluating the Effect of a COVID-19 Predictive Model to Facilitate Discharge: A Randomized Controlled Trial.

BACKGROUND: We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown. OBJECTIVES: The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS). METHODS: We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays. RESULTS: Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location. CONCLUSION: An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result. TRIAL REGISTRATION: ClinicalTrials.gov identifier: NCT04570488.

Validation of parsimonious prognostic models for patients infected with COVID-19.

OBJECTIVES: Predictive studies play important roles in the development of models informing care for patients with COVID-19. Our concern is that studies producing ill-performing models may lead to inappropriate clinical decision-making. Thus, our objective is to summarise and characterise performance of prognostic models for COVID-19 on external data. METHODS: We performed a validation of parsimonious prognostic models for patients with COVID-19 from a literature search for published and preprint articles. Ten models meeting inclusion criteria were either (a) externally validated with our data against the model variables and weights or (b) rebuilt using original features if no weights were provided. Nine studies had internally or externally validated models on cohorts of between 18 and 320 inpatients with COVID-19. One model used cross-validation. Our external validation cohort consisted of 4444 patients with COVID-19 hospitalised between 1 March and 27 May 2020. RESULTS: Most models failed validation when applied to our institution's data. Included studies reported an average validation area under the receiver-operator curve (AUROC) of 0.828. Models applied with reported features averaged an AUROC of 0.66 when validated on our data. Models rebuilt with the same features averaged an AUROC of 0.755 when validated on our data. In both cases, models did not validate against their studies' reported AUROC values. DISCUSSION: Published and preprint prognostic models for patients infected with COVID-19 performed substantially worse when applied to external data. Further inquiry is required to elucidate mechanisms underlying performance deviations. CONCLUSIONS: Clinicians should employ caution when applying models for clinical prediction without careful validation on local data.

An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department.

During the coronavirus disease 2019 (COVID-19) pandemic, rapid and accurate triage of patients at the emergency department is critical to inform decision-making. We propose a data-driven approach for automatic prediction of deterioration risk using a deep neural network that learns from chest X-ray images and a gradient boosting model that learns from routine clinical variables. Our AI prognosis system, trained using data from 3661 patients, achieves an area under the receiver operating characteristic curve (AUC) of 0.786 (95% CI: 0.745-0.830) when predicting deterioration within 96 hours. The deep neural network extracts informative areas of chest X-ray images to assist clinicians in interpreting the predictions and performs comparably to two radiologists in a reader study. In order to verify performance in a real clinical setting, we silently deployed a preliminary version of the deep neural network at New York University Langone Health during the first wave of the pandemic, which produced accurate predictions in real-time. In summary, our findings demonstrate the potential of the proposed system for assisting front-line physicians in the triage of COVID-19 patients.

A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients.

The COVID-19 pandemic has challenged front-line clinical decision-making, leading to numerous published prognostic tools. However, few models have been prospectively validated and none report implementation in practice. Here, we use 3345 retrospective and 474 prospective hospitalizations to develop and validate a parsimonious model to identify patients with favorable outcomes within 96 h of a prediction, based on real-time lab values, vital signs, and oxygen support variables. In retrospective and prospective validation, the model achieves high average precision (88.6% 95% CI: [88.4-88.7] and 90.8% [90.8-90.8]) and discrimination (95.1% [95.1-95.2] and 86.8% [86.8-86.9]) respectively. We implemented and integrated the model into the EHR, achieving a positive predictive value of 93.3% with 41% sensitivity. Preliminary results suggest clinicians are adopting these scores into their clinical workflows.

Prevalence and Outcomes of D-Dimer Elevation in Hospitalized Patients With COVID-19.

OBJECTIVE: To determine the prevalence of D-dimer elevation in coronavirus disease 2019 (COVID-19) hospitalization, trajectory of D-dimer levels during hospitalization, and its association with clinical outcomes. Approach and Results: Consecutive adults admitted to a large New York City hospital system with a positive polymerase chain reaction test for SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2) between March 1, 2020 and April 8, 2020 were identified. Elevated D-dimer was defined by the laboratory-specific upper limit of normal (>230 ng/mL). Outcomes included critical illness (intensive care, mechanical ventilation, discharge to hospice, or death), thrombotic events, acute kidney injury, and death during admission. Among 2377 adults hospitalized with COVID-19 and ≥1 D-dimer measurement, 1823 (76%) had elevated D-dimer at presentation. Patients with elevated presenting baseline D-dimer were more likely than those with normal D-dimer to have critical illness (43.9% versus 18.5%; adjusted odds ratio, 2.4 [95% CI, 1.9-3.1]; P<0.001), any thrombotic event (19.4% versus 10.2%; adjusted odds ratio, 1.9 [95% CI, 1.4-2.6]; P<0.001), acute kidney injury (42.4% versus 19.0%; adjusted odds ratio, 2.4 [95% CI, 1.9-3.1]; P<0.001), and death (29.9% versus 10.8%; adjusted odds ratio, 2.1 [95% CI, 1.6-2.9]; P<0.001). Rates of adverse events increased with the magnitude of D-dimer elevation; individuals with presenting D-dimer >2000 ng/mL had the highest risk of critical illness (66%), thrombotic event (37.8%), acute kidney injury (58.3%), and death (47%). CONCLUSIONS: Abnormal D-dimer was frequently observed at admission with COVID-19 and was associated with higher incidence of critical illness, thrombotic events, acute kidney injury, and death. The optimal management of patients with elevated D-dimer in COVID-19 requires further study.

An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department

During the COVID-19 pandemic, rapid and accurate triage of patients at the emergency department is critical to inform decision-making. We propose a data-driven approach for automatic prediction of deterioration risk using a deep neural network that learns from chest X-ray images, and a gradient boosting model that learns from routine clinical variables. Our AI prognosis system, trained using data from 3,661 patients, achieves an AUC of 0.786 (95% CI: 0.742-0.827) when predicting deterioration within 96 hours. The deep neural network extracts informative areas of chest X-ray images to assist clinicians in interpreting the predictions, and performs comparably to two radiologists in a reader study. In order to verify performance in a real clinical setting, we silently deployed a preliminary version of the deep neural network at NYU Langone Health during the first wave of the pandemic, which produced accurate predictions in real-time. In summary, our findings demonstrate the potential of the proposed system for assisting front-line physicians in the triage of COVID-19 patients.